\section{Developing a solution}
We introduce HypERLtrace as a prototype for understanding Erlang program behavior. It consists of two main components. First, the table view, for low-level trace analysis and second, a dynamic visualization for viewing process communication.

\subsection{Trace Analysis}
\begin{figure*}
\centering
\psfig{file=app.png, scale=0.65}
\caption{An example of an Erlang trace file opened using HypERLtrace. The black graph represents all the events, while the red represents the events found according to the search field.}
\label{fig:app}
\end{figure*}

%% The paragraph BELOW can be moved to problem space %%
Erlang trace files of large systems contain a lot of textual information. Even by abstracting some of the data, it can prove to be a task to read such files. One naive work around is tracing small portions of the program unique sets of trace flags. However, doing so may takes a long time and may require restarting the program multiple times. We introduce an alternative way, in that a user can trace their software once and filter the information afterwards without affecting further execution. 
%% The paragraph ABOVE can be moved to problem space %%

Our current prototype (Figure \ref{fig:app}) offers a variety of features that makes trace analysis of Erlang programs more convenient for the user. To load a trace file into the application, the user may use to following commands:

\begin{mylisting}
\begin{verbatim}
dbg:tracer(port, dbg:trace_port(file, "File.trace")).
dbg:p(self(), [timestamp, procs, m]).
dbg:p(new, [timestamp, procs, m]).
\end{verbatim}
\end{mylisting}

The process trace flag \texttt{new} can be used for programs that are started after the tracer has been set. This also hides interaction between system processes. On the other hand, for programs that require runtime tracing the flag \texttt{all} is more appropriate. No data is abstracted past the point of tracing.

The application is able to load the trace file and immediately fill in the event and process tables. The event table contains all the events sorted by time stamp. In addition, a graph is used to represent the number of events relative to their position in the trace. The process table is a tree structured table with expandable nodes to store information about spawning (such as module and function), as well as display statistics regarding sends, spawns and receives for each process in the trace. One of the main features of our prototype is the filtering ability. A user may type in keywords in the search field, and the application will not only display the events related to the search, but also highlight them on the graph. In addition, the lifetime of a specific process is displayed on the graph by simply hovering over the process name.
\begin{comment}
%not sure if this should be here or at the beginning of Problem space

We collected trace files of six large-scale Erlang systems.  The tracing was set up to look at new processes and the process related events  (send, receive, spawn and die) by using the trace functions: \texttt{dbg:p(self(), [m, procs, timestamp])} and \texttt{dbg:p(new, [m, procs, timestamp])}. 
\begin{enumerate}
\item{\emph{CouchDB}. Apache CouchDB is a distributed, fault-tolerant, nonSQL database server written in Erlang. The trace file was obtained by tracing a debugging shell with an Apache CouchDB database set up on localhost and running all of the provided test cases.}
\item{\emph{Yaws}. Yaws is a HTTP high performance webserver that is appripriate for hosting dynamic-content webapplications written in Erlang for lightweight concurrency. The trace was gathered by running the server on localhost and browsing through the pages.}
\item{\emph{Tsung}. Tsung is a distributed multi-protocol load testing tool for stress testing webservers supporting databases or otherwise. The trace file was generated by creating one hundred virtual clients trying to connect and chat on an Ejabberd server (see below).}
\item{\emph{Ejabberd}. Ejabberd is a cross-patform Jabber instant messaging server written in Erlang for multithreading ability. The trace file was generated by simulating a instant messaging conversation using Tsung.}
\item{\emph{Telephony Exchange Server}. Ericsson provided a runtime trace of a telephony exchange server that acts as a translator between the Ericsson proprietary protocol and H.248 protocol used by common media gateways.}
\item{\emph{Router}. Francesco Cesarini, a highly involved Erlang developer provided a runtime trace of an automated test case running on a router node. The router is part of the launching of a new wireless Instant Messaging gateway.}
\end{enumerate}


\end{comment}

\begin{figure*}
\centering
\psfig{file=arc1.png, scale=0.5}
\caption{Arc diagram representing an Ericsson Telephony Exchange Server acting as a translator between the Ericsson proprietary protocol and H.248 protocol used by media gateways}
\label{fig:arc1}
\end{figure*}

\subsection{Visualization}
An important feature of the application is a visualization component, that is capable of providing a high-level view of the interaction between concurrent processes (Figure \ref{fig:arc1}). We chose the arc diagram because its versatility in representing highly repetitive information in a large visual space. In addition, it enables the eye to detect visual patterns in the data \cite{wattenberg:arc}. To take the visualization one step further, we plan to add a highly interactive component for the user to be able to isolate and obtain information about communication patterns between a group of processes.

Our current version of the diagram has several features may prove useful to Erlang developers. To start off, processes are represented as squares, while communication is in the form of arcs. Processes that are the result of spawning, are kept next to their parents and lowered to show hierarchy. While too many arcs can become less visually appealing, we allow the user to hover over a square in order to allow the diagram to highlight all the communication from and to that particular process.

Much like all other visualization techniques, the issue of showing as much information as possible in the given space applies to the arc diagram. Although it is full screen, there is the possibility that there are simply too many processes and events to be able to fit on one screen. In order to combat this issue we introduced a few methods of data abstraction to the diagram. First, a ranking algorithm is used to determine the importance of each process
\footnote{Where $S(p)$ is the number of sends, $R(p)$ is the number of receives, $SP(p)$ is the number of spawns, $L(p)$ is the lifetime in $\mu$s of the process, $T$ is the total time in $\mu$s of the trace, and $1 \le C(p) \le n-1$ is the position p of found when it plays its first action relative start of the trace.}:
\begin{displaymath}     I(p) = 2\cdot S(p) + 1\cdot R(p) + 10\cdot SP(p)+ 2 \cdot (\frac {L(p)} T \cdot 100) + C(p) \end{displaymath} 
It is often the case that when looking at a high-level representation of large trace files, the user is mostly interested in the processes that are more involved than others. Our second method for conserving space is the collapsing of child processes by preserving their message sending patterns. Because parent and child nodes are kept together, it may be the case that for an important process, its children take up too much space. Therefore, we chose to collapse child processes with the same module, function and message sending pattern into a single square next to its parent. In future work, we plan to add a interactive way to collapse and expand processes depending on their properties. Furthermore, a time slider would be appropriate for simulating real-time concurrent communication.

\section{Preliminary Results}

The tool was tested on the different aforementioned large-scale Erlang systems. The visualization component yielded distinct arc patterns for each of the traces. In addition, we also noticed that there is a lot of communication to processes outside of the trace zone. This can be resolved by retracing the software with the \texttt{all} flag instead of \texttt{new}.

